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ABSTRACT 

This report describes a three-stage catalog use study 
conducted at the University of Wisconsin-Stout from 1982 through 1986 
to determine the types of searches conducted and search success rates 
for both the card and online catalogs during the library's transitic^. 
to the DataPhase Automated Library Information System (ALIS). Phase 
I, conducted before the introduction of online access, developed a 
baseline for future comparison; Phase II studied search type, search 
success, and use for both catalogs; and Phase III, conducted after 
the removal of the card catalog, concentrated on online catalog use. 
Data were collected by means of interviews with users for all three 
phases. Major findings include: (1) a majority of users performed 
subject searches rather than known-item searches; (2) virtually the 
only search types used were author, title, and subject; (3) use of 
the Library of Congress Subject Headings guide as part of the search 
strategy increased for online searchers; (4) use of online 
circulation information increased from Phase II to Phase III; (5) 
previous computer experience had no effect on users' success rates 
with the online catalog; and (6) Phase II users preferred the online 
catalog. A 16-item bibliography is provided, and 15 graphs depicting 
study results are appended. (KM) 
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ABSTRACT 



A three stage catalog use study was conducted at the 
University of Wisconsin — Stout, Menomonie, Wisconsin over 
the period 1982 to 1986. Data on types of searches 
conducted and search success rates were collected for both 
card catalog and online catalog use during the library's 
transition to the DataPhase Automated Library Information 
System (ALIS). Major findings included the following: 

A substantial majority of, users chose to 
use subject searches rather than known 
item searches at all stages of the survey. 

Traditional search types (author, title, 
subject) have remained virtually the only 
ones used inspite of the availability of 
new access points in the online catalog. 



Patrons' use of the Library of Congress Subject 
Headings guide (the **Red Books'*) as part of 
their search strategy in the online catalog 
increased from the 1984 to the 1986 study. Card 
catalog users, on the other hand, rarely 
used this tool. 

Use of the circulation information (showing 
whether the item sought was 'checked out, on the 
shelf, etc) in the online catalog increased 
from the second to the third stage surveys. 
Use of the bibliographic information declined. 

The factor of user's previous experience with 
computers had no effect on success rates in 
the online catalog. 

in the 1984 survey, when both card catalog and 
online catalog use were studied, users showed 
a preference for the online catalog. 
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Phase I 



A three phase catalog use study was conducted in 
order to analyze use patterns as the library changed from 
a traditional card catalog to an on-line catalog ♦ The 
purpose of the research was to identify ways that a 
particular clientele used a library catalog, noting 
changes over time. The primary goal was to evaluate the 
effectiveness of the online catalog in comparison to the 
card catalog in terms of user perceptions, of search 
success and ease of use* On a practical level it was 
hoped that the study would yield information about ways to 
enhance the catalog and teach its use in our library 
instruction program. The study's first stage in 1982 
measured use of the card catalog before introduction of 
any online access in order to establish baseline data for 
latex comparison with online catalog use statistics. 
Specific objectives were to identify the level of use of 
the various search opoions (known-item vs. subject 
searching), to determine search success rates and to 
measure the length of time spent for searches. 

Phase II 

A DataPhase ALIS II library system was installed in 
stages beginning in the summer of 1982. At the time of 
the second phase of the catalog use study in 1984 
circulation and cataloging functions were fully automated 
with all of the cataloged collections included in the 
file, retrospective conversion having been completed 
earlier. Ten terminals were available for the public to 
search using a search function called Inquiry. Inquiry is 
designed for staff use, having no online instructions or 
help screens and providing bibliographic information in 
the form of brief citations or the complete MARC format. 
Although the software for a user friendly public access 
version had not yet been developed, the library introduced 
this preliminary version of the online catalog to the 
university community through the bibliographic instruction 
program, written instructions, and staff assistance. Six 
terminals were located in the Reference area adjacent to 
the card catalog. The second phase of the study was 
administered to searchers at these terminals as rrell as 
card catalog users. In addition to repeating the questions 
about search type and success measured in the previous 
phase, the Phase II study also measured the relative 
quantity of searchers choosing each of the two catalogs 
and identified attitudes towards the ALIS catalog as well 
as problems encountered in using it. 



Phase III 



By the time of the thix'^d phase of the study in 1986, 
the evolution of the online catalog had brought major 
changes in catalog access at Stout. The ALIS Public 
Access Catalog (PAC) function, a search function with help 
screens, had been in use for about a year and the card 
catalog had been removed. In addition to the traditional 
author, title and subject search capabilities, the PAC 
opening menu offered searching by author/title and 
author/author combinations, call number, a limited form of 
boolean subject searching, and ISBN number. 

METHODOLOGY 

For the first phase of the study survey, questions 
were developed and a trial run was conducted to determine 
whether to use a written questionnaire or an interview 
technique. Results of the sample indicated that more 
accurate responses would be secured using interviews. An 
interview schedule was set up to include morning, 
afternoon, evening and weekend hours covering at least 
five da7« each week during a four week period in April, 
1982. Ir terviewers approached catalog users as they left 
the catalog. Subjects were asked if they would answer a 
few questions on their use of the catalog. The 
interviewers recorded their answers and any comments about 
the catalog or the library that were made during the 
interview . 

Tne two subsequent phases of the study, modelled on 
the Phase I design, were conducted at the same time of 
year and used the same information gathering technique. 
The questions asked in the Phase I questionnaire became 
the core of the survey each year and additional questions 
were added in the second two phases as staff identified 
specific questions about online catalog use that required 
investigation. The additional questions lengthened the 
interview sessions and caused a decrease in total 
respondents in each succeeding phase. 422 users 
participated in Phase I, 323 in Phase II (226 online 
catalog, 107 card catalog), and 289 in Phase III. Results 
were tabulated by the campus academic computing center 
using NPERC, a locally developed program that tallies 
frequency counts and percentages. Cross tabulations were 
made for the following variables: user statue (undei rad , 
graduate student, faculty or academic staff, other staff, 
visitor), type of search made (subject, author, title), 
method of introduction to the computer catalog (library 
instruction, written instructions, staff assistance, etc.) 
(second two phases), and prioi* experience with computers 
(third phase only). 
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RESULTS 



Data collected in the initial card catalog phase of 
the study indicated a preponderance of subject searching. 
'^6% of the searches of the total sample and 80% of 
searches conducted by undergraduates were subject 
searches. It was also found that subject searches took 
more time than known-item searches, with only 16% of the 
respondents reporting that their search was completed in 
less than one minute, as compared to 63% of known item 
searches. This finding could have implications for the 
number of terminals needed. 

Subject Searnhing 

The high proportion of subject searching was of 
particular interest because it was in this c.rea that the 
online catalog was expected to differ the most from the 
card catalog. First, in the card catalog, author, title, 
and subject entries were interfiled. The online catalog 
represented a change from a dictionary to a divided 
catalog in which searchers would have to specify what kind 
of search they wanted to do through the codes A for 
author, T for title, S for subject, etc. Limiting a 
search to the subject index would eliminate titles 
matching the search terms. Of course the title index 
could also be searched for a subject term, but this would 
be done as a conscious search strategy [and would 
eliminate an element of serendipity]. Second, two new 
search strategies, term truncation and the ability to 
search subject subdivisions as well as main headings would 
also be offered. Third, at this stage the online catalog 
lacked the see a'nd see also references found in the card 
catalog. In this situation, it was more important for 
users to know an exact Library of Congress subject heading 
than before and to be able to type it accurately. Errors 
in punctuation, spacing, or spelling were likely to yield 
a "No match found" message. 

Findings for the two subsequent phases of the study 
revealed a consistently high percentage of subject 
searching (Fig. 2). However, there were wide variations 
in the success rate of subject searching over the three 
phases. The percentage of subject searchers who reported 
finding no matches to their search term fluctuated from a 
high of 31% (1984 online catalog) to a low of 9% (1966 
online catalog) (Fig 3&4). Failure rates in the card 
catalog were more consistent; 16% in 1982 and 21% in 1984. 
The high failure rate for the online catalog in 1984 was 
probably caused by a combination of factors, i) eluding the 
relative inexperience of patrons in using onlii.e catalogs, 
the lack of an online see reference structure, and the 
chan^ from a dictionary to a divided catalog. Comments 
recorded in the 1984 survey reinforce the idea that many 
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suDject searches in the card catalog were actually 
performed in titles: 



There should be more subject headings 
othe^ than Library of Congress as in 
the card catalog. (Only Library of 
Congress headings were included in 
the card catalog). 

In the card catalog, I find Eiuch 
more information and I even use the 
same subject heading that I used in 
ALIS. 

There are more topics in the card 
catalo£C. 

Part of the sharp drop in subject search failures 
reported in the 1986 survey might be attributed to 
increased education of users in the use of the Library of 
Congress Subject Headings list, the "red books." A 
question asking subject searchers whether they had used 
this reference in their searching was added to the survey 
in 1984 and 1986 in an effort to measure the impact of the 
library instruction program in online catalog use. 
Instruction librarians made this a key point in both 
written and classroom instruction on online catalog use. 
Their efforts appear to have made a difference since 34% 
of ALIS subject searchers as opposed to only 4% of card 
catalog subject searchers reported using the LCSH in the 
1984 survey. The figure increased again to 42.5% in the 
1986 survey (Fig. 8) . 

Known- Ttem Search ing 

In addition to the library instruction effort, the 
factors of the new public access software and increased 
experience with the online catalog were likely 
contributors to the higher nuccess rates in the 1986 
survey. These elements would be expected to improve 
known-item searching as well as subject searching. The 
success rates of known-item searches did in fact s' 
considerable improvement in 1986, when the precentat of 
respondents reporting finding no match was 15%, 
corresponding closely to the figure of 17% for the card 
catalog in 1982 and down by almost half from the 1984 ALIS 
figure of 28%. 

The decline in search success rates, particularly for 
known-item searches, in the card catalog in 1984 as 
compared to 1982 (Fig. 4) was not anticipated. Some 
search failure can be attributed to collection failure 
rather than catalog f<ailure. That is, no items are found 
in response to a search term because the library does not 



own anything under the given heading. The representation 
of collection failure in th: ociinple might be expected to 
be higher than normal because 20% of the users reported in 
answer to another question that their reason for using the 
card catalog for this search was that they had already 
tried ALIS with no success. The change could also reflect 
the redirection of library resources toward the online 
catalog. The demands of implementation and maintenance of 
the online database had required closing the card catalog 
and filing cards for new titles in a separate supplemental 
catalog. Only a minimum of file maintenance was continued 
in the card catalog as cataloging personnel turned their 
efforts to online catalog needs. The library instruction 
program also discontinued education in card catalog use 
and concentrated its efforts on searching the online 
catalog . 

User Preference 

Since the 1984 phase of the study measured use of 
both the card catalog and on-line catalogs side-by--side , 
additional questions were asked to determine user 
preferences. Prior to conducting the survey, counts of 
users at the card catalog and online catalog were taken at 
random half hour periods. This count indicated that the 
relative traffic at the two catalogs was three online 
catalog users for each card catalog user. This preference 
for the online catalog was echoed by 79% of the ALIS 
respondents to the question "Which catalog do you prefer?" 
Forty percent of the ALIS users stated that they never 
used the card catalog anyraore. This finding was 
consistent with staff observations that many users, upon 
finding ALIS down on occasion, would postpone their 
catalog use until the online catalog was available rather 
than using the card catalog. Most frequently mentioned 
reasons for choosing ALIS were "ALIS is faster" and "ALIS 
is easier to use." It is interesting to note that users 
hud the impression that "ALIS is faster" although their 
responses to questions about how long their searches took 
actually showed that card catalog searches were completed 
in less time. 

The card catalog users included a significant number 
(42%) who had never used ALIS. Of those card catalog 
users who had used the online catalog, 58% preferred the 
card catalog, citing their familiarity with it a:id its 
ease of use most frequently. Thus each of the two 
catalogs had its own group of defenders and its core of 
users (interestingly, about 40% for each catalog) who used 
it exclusively. 
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Use of On-line C atalog Features 



There were also a few questions included in the l^dA 
and 1986 ALIS surveys to measure use of some of the 
special features offered to online searchers. Responses 
showed that the use of circulation status information, 
i.e. an online notice that the item searched is currently 
checked out or on the shelf, increased between 1984 and 
1986 (Fig. 10). 52% of the sample used this feature in 
1984 and 77% in 1986. In contrast, the number who 
reported using the bibliographic information screen 
decreased sharply. In both the 1984 and 1986 versions of 
search software, the complete bibliographic iniformation 
was accessed as an additional level of the search after 
brief citations had been viewed. The 1984 version of this 
screen was the MARC record with all field tags included. 
The 1986 version was a labelled bibliographic record 
showing only the information typically included on a card 
catalog card, except omitting the call number. (The call 
number was now found only on the circulation information 
screen.) In 1984, 33% reported using the bibliographic 
screen. In 1986, the figure was 7%. Since the 
bibliographic screen is the only place where the physical 
description, notes, and added entries including subject 
headings are displayed, this finding would be very 
interesting if it proved consistent over time and through 
future software modifications. The value of various 
e]^^ments of descriptive cataloging has been questioned in 
the past. If online catalog user behavior could provide 
data showing which information in catalog records is 
actually used, this could provide a meaningful rationale 
for changing cataloging practice. 

In 1986, an additional demographic question asking 
what previous experience the user had had with computers 
was included. The responses revealed that almost one- 
half, 49% of the sample, had no experience using computers 
other than the online catalog (Fig. 15). This group was 
compared with the total group and with subsets of users 
who had particular types of computer experience such as 
use of microcomputers or the campus academic computer. 
The results showed that there were no differences in 
levels of search success dependent on this variable. 

On-line CatAlog Problems 

Both the 1984 and 1986 online catalog surveys asked 
users what problems, if any, they had using the catalog. 
In both years the answer checked most often was "None." 
Fifty-nine percent reported having no problems in 1984 and 
45% in 1886. Of those who did report problems the most 
frequently mentioned were specific software "bugs*' 
(repeating screens in 1984, pageback malfunctions in 
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1986), mechanical difficulties with logging on and 
correcting typing errors, and subject searching. Subject 
searching problems were also mentioned in the general 
comments section of both surveys. The difficulties 
reported with subject searching confirmed the concerns of 
the library staff. At the time of both online catalog 
surveys, there were acknowledged authority control 
problems in the bibliographic file as the records had not 
yet been submitted to a complete AACR2 authority change. 
The lack of the see reference structure in the online 
catalog also added to the difficulty, of searching. These 
factors seemed especially troublesome in the subject file, 
where menus were long and the search sofi;ware often made 
it impossible to return to the menu screen after viewing a 
specific record. Subject searching is the area in which 
the online catalog seems most in need of enhancement. 
Automated authority control is in the developmental stages 
in most online systems. Some have addressed the problem 
by providing keyword access. 

Library Instruction 

In the absence of auhority control or keyword 
searching, our approach has been to emphasize subject 
searching in library instruction. The program apparently 
has helped users to overcome some subject searching 
problems. Cross tabulations showed that users who 
reported learning to use ALIS in a class by a librarian 
also reported higher rates of use of the Library o f 
Congress Subje ct Hftadtngs and lower failure rates for 
subject searching than the sample as a wliole. This 
subgroup also had a higher percentage reporting no 
problems in using ALIS (57% for those who learned to use 
ALIS in a librarian-taught class compared to 45% of the 
whole group and the 35% of those who learned on their own 
or from a friend) . 

While the library instruction program seemed to have 
an impact on searching, the effectiveness of online 
instructions is difficult to evaluate. It is interesting 
to note that the new access points (author-title and 
author-author combinations, boolean subject searching, 
call number searching) were n^t used by patrons although 
they are listed on the public access introductory screen, 
and help screens are provided for their use. This might 
be explained simply by the fact that users are not 
familiar with these options. Bui the use of the LCSH also 
had not been a standard part of search strategy in the 
card catalog and it did grow significantly, suggesting 
that the library's instruction program had a greater 
impact on search strategy than did online instructions. 
(For various reasons, the new access points were not 
taught as part of library instruction). 
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SjmilarltlftR with other catalog use sturfieR. 

Some of the results of this study were consistent 
with the findings of a number of other catalog use 
studies. The predominance of subject searching has been 
reported in a number of studies of on-line catalog use 
(t \rkey, p. 78). Because this is a departure from the 
findings of older studies of card catalog use, some have 
suggested that it is a peculiarity of on-line catalogs. 
However, in our case subject searching also predominated 
in the card catalog, both before and after the 
introduction of the online catalog. Unfortunately, 
catalog users also have been found to express more 
problems with subject searching than other kinds of 
searching (Markey, p. 78). Librarians at Stout expected 
user dissatisfaction with subject searching, perhaps at 
even higher levels than it appeared because of the "known 
bugs," cumbersome software, lack of a syndetic structure 
and limitations of the database (spelling, spacing, 
punctuation, etc.) that were unique to our particular 
catalog during the survey period. However, the findings 
of other studies show that our situation is common, 
regardless of the type of online catalog involved. Markey 
reported that subject searching was difficult for users on 
all systems studied at six f ocused-group interview sites 
(p. 82). These catalogs featured different combinations of 
keyword access, online instructions, term truncation, and 
other search aids. One factor that increased success in 
subject searching at Stout was the use of the LCSH as part 
of the search strategy. 

Users preference for online over card catalogs has 
been widely reported (Markey, Moore, Kranich). At Stout, 
this preference was demonstrated in the second phase of 
the study when card catalog and online catalog users were 
both surveyed. Preference for the online catalog was 
demonstrated both by users "voting with their feet" in 
choosing to use the online catalog by a 3 to 1 margin and 
in stating this explicitly in answer to a survey question. 



T.-in^i tatinnR of thet fitudv and eugffe Btions for fu r_thiSr 

research. 

The measure of subject search success in terms of the 
number of citations retrieved gives an incomplete picture 
of a complex transaction. Two dimensions of this process 
that were not adequately measured by the study are 
precision, that is, the appropriateness of specific 
citations retrieved, and the element of time. 

One of the questions asked in each stage of the study 
was how long l irches took (Fig 5-7). Data collected 
showed the predictable finding that subject searching was 
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considerably more time-consuming than known-^item 
searching. This was true in both the card catalog and 
online catalog and was consistent throughout the three 
study stages. Although the question was designed to 
reveal how long it took a patron to complete a search on a 
single search term, the researchers felt that some 
respondents answered in terms of the total search sespion, 
during which they may have entered several related search 
terms- In the 1986 survey, the question was divided into 
two parts in oruer to clarify this point. However, there 
was still some doubt whether the question was understood. 
In the online catalog, a search on a single term can take 
several discrete steps, each requiring separate 
keystrokes. Strategy can be modified as the cearch 
proceeds. This dynamic quality makes it difficult to 
distinguish when one search ends and another begins during 
any given session. The question of length of time of 
searches has implications for queuing and the number of 
terminals needed for patrcn use. This information can 
probably be obtained more effectively by queuing studi<5S. 
Two studies of terminal use were conducted at Stout during 
1983 and 1984. These investigations measured the average 
number of users per hour for each terminal in the library, 
the total number of users at each terminal, and the number 
of users at the card catalog. The data idf»ntitied th^ 
terminals with the highest and lowest use, showed 
increased use of terminals from 1983 to 1984, and revealed 
a higher percentage of patrons choosing the online catalog 
over the card catalog than the 3 to 1 ratio reported in 
this study. 

Another limitation of this study me*:hod is the 
reliance on the memories and candor of people questioned. 
There may be a tendency on the part of people questioned 
to give anawers that they thlrk will please the 
interviewer or demonstrate their own competence. There is 
also no measure of the reasons for search failure. 
Searches that result in "no match" could be the result of 
spelling errors, inadequate subject headings, or the lack 
of material in the collection, just to name a few 
potential problems. The success rate of searching is 
actually a measure of a composite picture of user skill, 
catalog effectiveness, and collection adequacy. One way 
to obtain more specific information about catalog searches 
is through transaction log analysis (Stout's ALIS system 
does not have this capability). This automatic monitoring 
of searches would provide data to measure keying errors, 
spelling errors, the types of searches, number of matches, 
amount of use of online help screens, and other purely 
quantitative factors . 
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SUMMARY 



Through the four year transition period from a 
dictionary card catalog to an online public access 
catalog, the types of searches and relative percentage of 
each remained consistent, with subject searching 
predominant. New search strategies such as call number 
searching, ' tthor-title combination searching and boolean 
subject searching have not been adopted in significant 
numbers. However, the new features of online circulation 
information and online instructions are used. 

Online catalog search success rates were not as high 
as those of the card catalog initially, but improved over 
time, to the point whe re the highest search success rates 
were recorded in the most recent online catalog test 
phase. When both the card catalog and online catalog were 
available for use, patrons expressed preference for the 
online catalog. While acceptance of the online catalog 
has been high, dissatisfaction with subject searching has 
been expressed. A concerted effort by the library 
instruction program to teach the use of the Library "^f 
Congregfi Subject Headings had a positive impact, but the 
lack of a reference structure and software problems with 
subject searching remain. 
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Appendix 



In all stages of the study, undergraduates made up more 
than 75% of the survey sample. The numbers of other categories 
of users were not large enough to obtain meaningful data on 
how their catalog use differed from that of undergraduates. 



User status 1982 CCC) User status 1984 (CO 




n undergrad 
19 grad 



B fac/staff 
□ other staff 
n vis i tor s 



User status 1984 (ALIS> User status 1986 (ALIS) 




Type 
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The high proportion of subject searching remained 
constant throughout all survey phases. 
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Initially, online catalog users reported less search success 
than card catalog users. However, the online catalog figures for 
the most recent survey show the highest levels of search success 
as measured by number of citations matching the search terra • 

Subject sear-ch - No m^ftolies 



4f1 1 




1902 1904 1984 190G 
year 

Search success may also be shown by the measure of its opposite, 
search failure. The level of searches yielding no matches increased 
in both catalogs in 1984, but decreased to a low in the online catalog 
in 1986. 

J8 





CC CC ALIS ALIS 



1932 1904 1984 1986 
year* 
(Fig. 4) 

Known item search success patterns paralleled those seen 
for subject searching. The transition year of 1984 showed declines 
in search success, but 1986 performance showed the highest levels 
of success. 
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(Fig. 5) 

Leiigtli (time) of searches for subject 




□ 6-18 rriin. 

□ 1Q+ min. 



Subject searching proved to be more time- 
consuming than known-item searching in all 
survey stages ♦ The category for over 10 
minutes was added in 1984 and 1986 after 
results from 1982 suggested that a further 
breakdov/n at the high end of the scale was 
needed. Part of the decline in time spent 
on subject searches in 1986 was due to the 
addition of a question asking the length of 
the total search session as opposed to the 
subject of this question, which was the 
time required to search a particular term. 
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This g.^aph represents the percentages of subject searches 
that took 6 minutes or longer to complete. See explanation for 
figure 5 for further comment. 
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Over 85% of knovm item searches in the online catalog 
were completed in 3 minutes or less. The longer search times 
reported in 198A may be partly due to the fact that the survey 
did not explicitly distinguish between a -single search and 
the total search session in that year. 
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The importance of using the proper vocabulary in online subject 
searching was emphasized in the bibliographic instruction program. 
The message had an impact on user's search behavior. Card catalog 
users did not use the LCSH to a comparable extent. 
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This graph shows the ways patrons learned to use the on-line 
catalog. They were allowed to check ail answers that applied, 
so the total number of responses is greater than the total 
number of people surveyed. In 1984, a one page instigation 
sheet as v>cll as a detailed manual was available at each terminal. 
These were the most heavily used form of instruction at that time. 
By 1986 more people had been reached in classes pre'^ented by the 
library instruction program. The highest number of responses was 
in the category of experimentation on own. This reflects the 
heavy use of on-line instructions included as part of the more 
user friendly software version called PAC, (Public Access Catalog) 
which was in operation in 1986. 
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An increase in use of the circulation information provided 
by the online catalog was registered over time. 
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1986 respondents indicated that they used the circulation 
information screen most heavily. This screen includes the call 
number. The high use of the menu screen, which is a short 
citation, may represent an error in question interpretation 
on the part of the interviewers. 
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About 40% of 1986 ALIS users indicated that they had used 
online HELP instructions at some time in their use of ALIS 
(not necessarily for the current search). 
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The surveys were conducted at terminals in the Reference 
area of the library. The majority of users had also used terminals 
in other " -catijns of the library. 
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The percentage of users who indicated that they had ever 
taken advantage of the availability of a printer with ALIS 
remained the same over time. 
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49% of 1986 ALIS users had no previous experience with 
other computers. This variable had no impact on their 
searching success. 
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